Image processing method, apparatus and memory medium therefor

ABSTRACT

There is provided an image combining apparatus comprising an image input unit for entering photographed images, an image memory for temporarily storing the entered images, a corresponding point extracting unit for extracting the corresponding point information between the images stored in the image memory, a photographing method discriminating unit for discriminating the photographing method at the photographing operation based on the extracted corresponding point information, a combination process unit for appropriately combining the images according to the discriminated photographing method, and a combined image memory for storing the combined image. The corresponding point extraction unit extracts the corresponding points between two images, and the photographing method discrimination unit discriminates whether the translational photographing method or the panning photographing method is used. The combination process unit executes combination of the images, based on the discriminated photographing method.  
     The invention also provides an image processing method capable of automatically selecting the image combining method, solely from the images.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to an image combining method forcombining partially overlapping plural images to generating a panoramicimage or a high definition image, and an apparatus and a memory mediumtherefor.

[0003] 2. Related Background Art

[0004] In case of generating for example a panoramic image by combiningplural images, it is important to select an appropriate combiningprocess according to the photographing method. The combining processmentioned above is for example an affin conversion, a projectionconversion, a cylindrical mapping conversion etc.

[0005] Conventionally, the photographing method is either entered by theoperator at the combining operation, or estimated by the outputinformation of a hardware such as gyro, mounted on the camera at thephotographing operation.

[0006] In the former method, however, the operator is required to enterthe photographing method at the combining operation and has therefore toremember the situation of the photographing operation. On the otherhand, the latter method is associated with a drawback that the cameraitself becomes bulky since a detector such as a gyro has to be mountedon the camera. Also, in case of an off-line combining operation, theoutput of the gyro has to be retained by a suitable method.

SUMMARY OF THE INVENTION

[0007] In consideration of the foregoing, an object of the presentinvention is to provide an image processing method utilizing images andcapable of automatically selecting the combining method.

[0008] Another object of the present invention is to use an image fordiscriminating the photographing method and to effect image combinationaccording to the photographing method, thereby achieving imagecombination of a higher definition, depending on the photographingmethod.

[0009] Still another object of the present invention is to evaluate theimage combination based on at least one of plural combining methods,thereby selecting an appropriate combining method.

[0010] Still another object of the present invention is to extract acorresponding point in an overlapping area of plural images, therebyselecting the combining method in more exact manner.

[0011] Still another object of the present invention is to enable easierdiscrimination of translation displaced photographs or pannedphotographs by employing cylindrical mapping conversion or sphericalmapping conversion.

[0012] Still another object of the present invention is to enableautomatic formation of a panoramic (for 360° around) image with a higherprecision.

[0013] An image processing method according to an aspect of the presentinvention comprises an input step of entering plural images, and aselection step of selecting an appropriate combining method from pluralcombining methods, utilizing the plural image information.

[0014] Still other objects of the present invention, and the featuresthereof, will become fully apparent from the following detaileddescription of the embodiments, to be taken in conjunction with theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015]FIG. 1 is a block diagram showing the configuration of an imagecombining apparatus of a first embodiment;

[0016]FIG. 2 is a view showing a translation photographing mode in caseof photographing for example an original image;

[0017]FIG. 3 is a view showing a panning photographing mode in case ofphotographing for example a landscape;

[0018]FIG. 4 is a flow chart showing an image combining sequence of theimage combining apparatus;

[0019]FIG. 5 is a view schematically showing a corresponding pointextracting process;

[0020]FIGS. 6A and 6B are views schematically showing correspondingpoints in images obtained in translation photographing and panningphotographing;

[0021]FIG. 7 is a flow chart showing a photographing methoddiscriminating sequence on a photographing method discriminating unit102;

[0022]FIGS. 8A and 8B are views showing a combination process bydeveloping a combination image memory 105;

[0023]FIG. 9 is a block diagram showing the configuration of an imagecombination apparatus of a second embodiment;

[0024]FIG. 10 is a view showing a cylindrical mapping conversion processin an image conversion unit 109;

[0025]FIG. 11 is a view schematically showing an image after conversion;

[0026]FIG. 12 is a flow chart showing a photographing methoddiscriminating sequence in a photographing method discrimination unit107;

[0027]FIG. 13 is a view showing a combined image in case panningphotographing is identified;

[0028]FIG. 14 is a block diagram showing an image combination apparatusof a third embodiment;

[0029]FIGS. 15 and 16 are flow charts showing an image combiningsequence in the third embodiment;

[0030]FIG. 17 is a view showing n combined images after parameterconversion;

[0031]FIG. 18 is a view showing the viewing angles in the respectiveimages;

[0032]FIG. 19 is a view showing a center line S of the overlapping areaof images;

[0033]FIG. 20 is a view showing combined images prior to conversionprocess;

[0034]FIG. 21 is a view showing combined images after conversionprocess;

[0035]FIG. 22 is a view schematically showing the conversion process;

[0036]FIG. 23 is a view schematically showing generated combined images;

[0037]FIG. 24 is a view showing generation of an image by 360°cylindrical mapping;

[0038]FIG. 25 is a block diagram of a computer system realizing an imagecombining apparatus of a fourth embodiment; and

[0039]FIG. 26 is a flow chart showing an image combining sequence in thefourth embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040]FIG. 1 is a block diagram showing the configuration of an imagecombination apparatus of a first embodiment of the present invention,wherein shown are an image combination process unit 1, an image inputunit 100 for entering photographed images, an image memory 104 fortemporarily storing the input images, and a corresponding pointextraction unit 101 for extracting the corresponding point informationamong the images stored in the image memory 104.

[0041] There are further provided a photographing method discriminationunit 102 for discriminating the photographing method at thephotographing operation, based on the information of the extractedcorresponding points, a combination process unit 103 for appropriatelycombining the images according to the discriminated photographingmethod, and a combined image memory 105 for storing the combined images.

[0042] The photographing method discriminated by the photographingmethod discrimination unit 102 includes, for example, translationphotographing in which the photographs are taken by varying thephotographing position without changing the photographing direction, andpanning photographing in which the photographs are taken by varying thephotographing direction without changing the photographing position.FIG. 2 shows translation photographing in case of photographing forexample original images. After the photographing at a position 1, thecamera is moved to a position 2 for photographing again. The movementfrom the position 1 to 2 involves changes in the main parameters oftranslation movement (Δx, Δy), rotation (θ) and magnification differencem. Such photographing mode will be called translation photographing.

[0043]FIG. 3 shows the panning photographing in case of photographingfor example a landscape. The photographing operations are executed withthe camera in positions 1, 2 and 3. Main changes involve rotation (ψ, φ,θ) about the X, Y, Z axes. In particular, the rotation φ about the Yaxis is a governing factor since the photographing operations areexecuted in succession while the camera is rotated in the horizontaldirection. Such photographing mode will be called panning photographing.

[0044] The photographing method discrimination unit 102 discriminates,in the present embodiment, whether the photographing method istranslation photographing or panning photographing.

[0045] In the following there will be explained the function of theimage combining apparatus of the above-described configuration. FIG. 4is a flow chart showing the image combining sequence of the imagecombining apparatus. At first the image input unit 100 enters image data(step S1). The input image data consists of an image photographed with acamera and digitized by A/D conversion, but there may also be entered animage stored in a recording medium or a memory device.

[0046] The image entered by the image input unit 100 is temporarilystored in the image memory 104 (step S2). Then there is discriminatedwhether two images have been entered (step S3). If two images have beenentered, the entered two images are used for executing a process ofextracting corresponding points (step S4). FIG. 5 schematically showsthe corresponding point extracting process. The corresponding points areextracted in the following manner, utilizing the images 200L, 200R inthe image memory 104:

[0047] 1) There is set a template 201 of p×p pixels, with the center ata point P1L on a reference image 200L.

[0048] 2) A correlation value Zmn is generated according to thefollowing equation (1) at each moved point, while the template 201 ismoved within a search area 202 set on a reference image 200R:$\begin{matrix}{z_{m\quad n} = \frac{\sum\limits_{x}{\sum\limits_{y}{{f_{L}\left( {x,y} \right)} \cdot {f_{R}\left( {{x - m},{y - n}} \right)}}}}{\sqrt{\sum\limits_{x}{\sum\limits_{y}{f_{L}\left( {x,y} \right)}^{2}}}\sqrt{\sum\limits_{x}{\sum\limits_{y}{f_{R}\left( {{x - m},{y - n}} \right)}^{2}}}}} & (1)\end{matrix}$

[0049] wherein Σ_(x)Σ_(y) indicates summation on all the pixels in thetemplate.

[0050] 3) Correlation values Zmn on the moved points on the referenceimage 200R are compared to extract a point providing the maximum value.

[0051] 4) For the point P1L (i, j), the coordinate (i′, j′) of thecorresponding point is given by i′=i−m′, j′=j−n′ wherein (m′, n′)indicates the movement amount providing the maximum correlation value.

[0052] The corresponding point extraction unit 101 sets plural templateson the image 200L shown in FIG. 5 and extracts the corresponding pointfor each template, according to the procedure 1) to 4) explained above.The extracted corresponding points are stored in an unrepresentedmemory.

[0053] Based on the corresponding point information extracted in thestep S4, the photographing method discrimination unit 102 discriminatesthe photographing method (step S5). In the following there will beexplained the discrimination of the photographing method.

[0054]FIGS. 6A and 6B schematically show the corresponding pointsbetween the images obtained by translation photographing and panningphotographing. The coordinate relationship between the correspondingpoints shown in FIG. 6A can be described by affin conversion representedby the equations (2):

x′=Ax+By+C

y′=−Bx+Ay+D  (2)

[0055] wherein (x, y) is the coordinate on the image 300L in FIG. 6A,while (x′, y′) is the coordinate on the image 300R. Also the coordinaterelationship between the corresponding points shown in FIG. 6B can bedescribed by projection conversion represented by the followingequations (3): $\begin{matrix}{{x^{\prime} = \frac{{m_{1}x} + {m_{2}y} + m_{3}}{{m_{7}x} + {m_{8}y} + 1}}{y^{\prime} = \frac{{m_{4}x} + {m_{5}y} + m_{6}}{{m_{7}x} + {m_{8}y} + 1}}} & (3)\end{matrix}$

[0056]FIG. 7 is a flow chart showing the photographing methoddiscriminating sequence in the photographing method discrimination unit102. At first the corresponding point information obtained in thecorresponding point extraction unit 101 is read from a memory which isnot shown (step S11).

[0057] The read corresponding points are subjected to calculations bythe least square method, based on the conversions represented by theforegoing equations (2) and (3), for calculating a combination error forevaluating the quality of image combination (step S12). The evaluatingequation by the least square method is shown in the following. In caseof the affin conversion, there are calculated parameters A, B, C and Dfor minimizing an error E2-of the equation (4):

E _(2=Σ[{) x′−(Ax+By+C)}² +{y′−(−Bx+Ay+D)}²]  (4)

[0058] Also in case of the projection conversion, there are calculatedparameters m₁ to m₈ for minimizing an error E3 of the following equation(5): $\begin{matrix}{\begin{matrix}{{E3} = \quad {\sum\left\lbrack {\left\{ {{x^{\prime}\left( {{m_{7}x} + {m_{8}y} + 1} \right)} - \left( {{m_{1}x} + {m_{2}y} + m_{3}} \right)} \right\}^{2} +} \right.}} \\{\quad \left. \left\{ {{y^{\prime}\left( {{m_{7}x} + {m_{8}y} + 1} \right)} - \left( {{m_{4}x} + {m_{5}y} + m_{6}} \right)} \right\}^{2} \right\rbrack}\end{matrix}\quad} & (5)\end{matrix}$

[0059] The error E2 obtained in the step S12 is compared according to arelation (6) (step S13):

E2<Th  (6)

[0060] Thus, if the error E2 is smaller than a predetermined thresholdvalue Th, there is identified the affin conversion, namely thetranslation photographing method (step S15). On the other hand, if theerror E2 is at least equal to the predetermined threshold value Th, theerror E2 is compared with the error E3 according to a relation (7):

E2≦E3  (7)

[0061] Thus, if the error E2 does not exceed the error E3, there isidentified the affin conversion or namely the translation photographing(step S15), but, if the error E3 is smaller than the error E2, there isidentified the projection conversion, namely the panning photographing(step S16). The result of discrimination is given, together with theparameters, to the combination process unit 103.

[0062] Then a combination process is executed, based on the result ofdiscrimination in the step S5 (step S6). FIGS. 8A and 8B illustrate acombination process, showing the development of a combination imagememory 105. FIG. 8A shows the combination process in case thetranslation photographing is discriminated, while FIG. 8B shows that incase the panning photographing is discriminated.

[0063] The combination process unit 103 executes the combination processin the following sequence:

[0064] 1) A joint position S between the images is set, based on thecombination parameters.

[0065] 2) The image 200L is read from the image memory 104 and the areaat the left side of the joint S is written into the combination imagememory 105.

[0066] 3) The area at the right side of the joint S is written into thecombination image memory 105, based on the image 200R. The coordinate(x′, y′) on the image 200R, corresponding to a point P1 (x, y) in FIG.8A is calculated by the equations (2). Also the coordinate on the image200R, corresponding to a point P2 (x, y) in FIG. 8B is calculated by theequations (3).

[0067] 4) The pixel value at the corresponding coordinate (x′, y′) onthe image 200R is written as the pixel value at the point P1 (x, y) orP2 (x, y). The combination process is executed in this manner.

[0068] In case the corresponding coordinate value (x′, y′) in theforegoing sequence step 3) contains a decimal fraction, the pixel valuemay be generated by a known interpolating process.

[0069] Through the sequence S1 to S6 in FIG. 4, the combined image isstored in the combination image memory 105 as shown in FIGS. 8A and 8B.

[0070] As the image combination apparatus of the first embodimentdiscriminates the photographing method solely from the images andexecutes the optimum combination process according to the result ofdiscrimination, the user need not remember the photographing method anda high-quality combined image can be constantly obtained.

[0071] Even when the combination method does not correspond to thephotographing method, the image combination is executed by suchcombination method if the combination error is sufficiently small, andthe obtained result is completely acceptable since the appropriatecombination method is selected also in such case.

[0072] Also in the generation of the combination parameters etc., it isalso possible to calculate the correlation value of the overlapping areaby displacing an image by a pixel at a time, or deforming an image, withrespect to the other image.

[0073] The process time can be reduced by extracting the correspondingpoints and utilizing the information of such corresponding points, sincethe calculation becomes simpler.

[0074]FIG. 9 is a block diagram of an image combination apparatus of asecond embodiment of the present invention, wherein components same asthose in the first embodiment are referred to by same numbers and willnot be explained further.

[0075] The image combination apparatus of the second embodiment isfeatured by a focal length input unit 106, an image conversion unit 109,a photographing method discrimination unit 107 and a combination processunit 108.

[0076] The focal length input unit 106 serves to enter the focal lengthat the photographing operation. The focal length may be entered by theuser either by the input of a numerical value through a keyboard or bythe selection from plural values, or by recording the focal lengthinformation together with the image at the photographing operation andreading the recorded information by the focal length input unit 106.

[0077] The image conversion unit 109 converts the image data utilizingthe focal length, based on the cylindrical mapping conversion.Corresponding point extraction units 1010, 1011 extract thecorresponding points between the images, according to a process similarto that in the first embodiment.

[0078] The photographing method discrimination unit 107 discriminates,as in the first embodiment, whether the photographing method istranslation photographing or panning photographing, but employedevaluation method is different from that of the first embodiment. Thecombination process unit 108 executes image combination according to thediscriminated photographing method.

[0079] In the following there will be explained the function of theimage combination apparatus of the second embodiment, having theabove-described configuration, wherein steps similar to those in theforegoing first embodiment are numbered same and will not be explainedfurther.

[0080] The image conversion unit 109 executes cylindrical mappingconversion on the image data stored in the image memory 104, based onthe focal length obtained by the focal length input unit 106. FIG. 10shows the cylindrical mapping conversion process in the image conversionunit 109.

[0081] In FIG. 10, there are shown an original image 200 read from theimage memory 104, and an image 201 after the cylindrical mappingconversion. The coordinate (x, y) before the cylindrical mapping isperformed and the coordinate (φ, v) after the cylindrical mapping isperformed are correlated by the following equations (8): $\begin{matrix}{{x = {f\quad t\quad a\quad n\quad \varphi}}{y = {\frac{\sqrt{x^{2} + f^{2}}}{f}v}}} & (8)\end{matrix}$

[0082] The image conversion unit 109 converts the image according to theequations (8). FIG. 11 schematically shows the image after conversion.In the image combination process unit 2 of the image combinationapparatus of the second embodiment, the corresponding point extractionunit 1010 executes extraction of the corresponding points between theoriginal images, while the corresponding point extraction unit 1011executes extraction of the corresponding points utilizing the imagesconverted by the cylindrical mapping conversion. Consequently there aregenerated two sets of paired corresponding points.

[0083] The photographing method discrimination unit 107 discriminatesthe photographing method, utilizing the corresponding point informationgenerated by the corresponding point extraction units 1010, 1011.

[0084]FIG. 12 is a flow chart showing the photographing methoddiscrimination sequence in the photographing method discrimination unit107. At first the corresponding point information obtained in thecorresponding point extraction units 1010, 1011 is read from a memorywhich is not shown (step S21).

[0085] Thus read corresponding points are used in the calculations bythe least square method, based on the foregoing conversion equations (2)(step S22).

[0086] The equations (2) alone are employed in order to effect affincoordinate conversion on the image obtained by cylindrical mappingconversion, even in case of the image taken by the panningphotographing. Stated differently, an appropriate combination method canbe selected by calculating the combination error by the equations (2) onthe image subjected to cylindrical mapping conversion and the image notsubjected to such cylindrical mapping conversion.

[0087] Equations (9) show the evaluation by the least square method.More specifically, there are calculated parameters A_(o), B_(o), C_(o),D_(o), A_(s), B_(s), C_(s), D_(s) minimizing the errors E4, E5 of theequations (9), on the paired corresponding points (x_(o),y_(o))−(x_(o)′, y_(o)′) obtained by the extraction from the originalimages and those (x_(s), y_(s))−(x_(s)′, y_(s)′) obtained by theextraction from the images subjected to cylindrical mapping conversion:$\begin{matrix}{\begin{matrix}{{E4} = \quad {\sum\left\lbrack {\left\{ {x_{O}^{\prime} - \left( {{A_{O}x_{O}} + {B_{O}y_{O}} + C_{O}} \right)} \right\}^{2} +} \right.}} \\{\quad \left. \left\{ {y_{O}^{\prime} - \left( {{{- B_{O}}x_{O}} + {A_{O}y_{O}} + D_{O}} \right)} \right\}^{2} \right\rbrack} \\{{E5} = \quad {\sum\left\lbrack {\left\{ {x_{S}^{\prime} - \left( {{A_{S}x_{S}} + {B_{S}y_{S}C_{S}}} \right)} \right\}^{2} +} \right.}} \\{\quad \left. \left\{ {y_{S}^{\prime} - \left( {{{- B_{S}}x_{S}} + {A_{S}y_{S}} + D} \right)} \right\}^{2} \right\rbrack}\end{matrix}\quad} & (9)\end{matrix}$

[0088] Then comparison is made whether the error E4 obtained in the stepS22 is smaller than a threshold value Th (E4<Th) (step S23). If theerror E4 is smaller than the predetermined threshold value Th, there isidentified the translation photographing and there is selected acombination method corresponding to the translation photographing (stepS25).

[0089] If the error E4 is at least equal to the threshold value Th,comparison is made whether the error E4 exceeds the error E5 (E4≦E5)(step S24). If the error E4 does not exceed the error E5, there isidentified the translation photographing (step S25), but, if the errorE5 is smaller than E4, there is identified the panning photographing(step S26), and, after the selection of a combination methodcorresponding to the result of such discrimination, the sequence isterminated. The result of discrimination is given, together with theparameters, to the combination process unit 108.

[0090] The function of the combination process unit 108 is basicallysame as that in the foregoing first embodiment. It is however differentfrom the first embodiment in that, in case the panning photographing isidentified, the photographing method discrimination unit 107 reads theoriginal images from the image memory 104, then converts the images bycylindrical mapping conversion by a process similar to that in the imageconversion unit 109 and writes the pixel values in the combined imagememory 105 utilizing the above-mentioned parameters A_(s), B_(s), C_(s),D_(s).

[0091]FIG. 13 shows a combined image in case the panning photographingis identified. The cylindrical mapping conversion is executed again atthe image combination, but it is also possible that the image conversionunit 109 stores the combined image in the image memory 104 and thecombination process unit 108 executes the image combination by readingthe stored image after cylindrical mapping conversion.

[0092] It is to be noted that a better image may be obtained by theprojection conversion as in the first embodiment, if the image angle ofthe combined image obtained from the panning photographing is not toolarge.

[0093] In such case, the image combining method may be switchedaccording to the image angle, after the discrimination of the panningphotographing.

[0094] Also, in the panning photographing operation explained in theforegoing, it is assumed that the photographing direction is changedonly in one direction (for example lateral direction), but, if thephotographing direction is changed in two or more directions (forexample lateral and vertical directions), the known spherical mappingconversion is more preferable and the image combining method may beswitched also in such case.

[0095]FIG. 14 is a block diagram showing the configuration of an imagecombination apparatus of a third embodiment of the present invention,which is featured in executing a process corresponding to thecombination of n images. The image combination process unit 1 in thethird embodiment is similar in configuration to that in the firstembodiment.

[0096] Referring to FIG. 14, an n-image integrating unit 111 generatesand retains various information for combining n images. A focal lengthestimation unit 112 estimates the focal length. Consequently the thirdembodiment is applicable also in case the focal length is unknown.However, if the focal length is already known, input means may beseparately provided for entering the value of the focal length. Thereare also provided a main image memory 113 for storing information of nimages, and an n-image integrating unit 114 for generating a panoramicimage by combining n images.

[0097]FIGS. 15 and 16 are flow charts showing an image combiningsequence in the third embodiment. At first two images are read from themain image memory 113 and supplied into the image combination processunit 1 (step S31). It is assumed that the number of images is stored, bymeans not shown, in the n-image integrating unit 111.

[0098] The image combination process unit 1 combines two images in aprocess similar to that in the foregoing first embodiment (step S32).The combined image retained in the combined image memory 105 may bedisplayed in display unit not shown, in order that the user canconfirmed the result of combination. The coordinate data of thecorresponding points between the images, the combining parameters andthe information on the photographing method are retained in the n-imageintegrating unit 111.

[0099] Then there is discriminated whether combination of n images hasbeen completed (step S33). If not, the sequence returns to the step S31to repeat the sequence, but, if completed, the results of (n−1)discriminations, retained in the n-image integrating unit 111, arechecked and the photographing method for the n images is determined bymajority as the translation photographing or the panning photographing(step S34).

[0100] The result of discrimination of the photographing methoddetermined in the step S34 is identified (step S35). More specifically,if the translation photographing is discriminated, the information onthe corresponding points among the images, retained in the n-imageintegrating unit 111, is read in succession and the affin conversionparameters A_(i), B_(i), C_(i), D_(i) are calculated by the least squaremethod according to the equations (2) (step S36).

[0101] Then thus calculated affin conversion parameters are convertedinto combination parameters A_(i)′, B_(i)′, C_(i)′, D_(i)′ based on theimage 1 (step S37). FIG. 17 shows the combination of n images after theconversion of the combination parameters.

[0102] The n-image combination unit 114 uses the converted combinationparameters A_(i)′, B_(i)′, C_(i)′, D_(i)′ to generate the pixel value ofeach point based on the coordinate conversion of the equations (2) andstores such pixel values in the main image memory 113 (step S38). Themethod of generating the pixel value is similar to that in the foregoingfirst embodiment.

[0103] On the other hand, if the step S35 discriminates the panningphotographing, there is displayed, on display unit not shown, a messagefor confirming a 360° combination (step S39), and there is discriminatedwhether 360° combination is instructed (step S40). If 360° combinationis not instructed, the focal length estimation unit 112 estimates thefocal length in the following manner (step S41).

[0104] As shown in FIG. 3, the panning photographing operation isexecuted by changing the viewing field by the rotation (ψ, φ, θ) aboutthe X, Y, Z axes. Consequently, for converting the coordinate betweenthe images, there is given a matrix M represented as (10):

M=FRF ⁻¹

[0105] wherein: $\begin{matrix}{{R = \begin{bmatrix}{\cos \quad {\psi cos\varphi}} & {{\sin \quad {\psi cos\theta}} + {\cos \quad {\psi sin\varphi sin\theta}}} & {{\sin \quad {\psi sin\theta}} - {\cos \quad {\psi sin\varphi cos\theta}}} \\{{- \sin}\quad {\psi cos\varphi}} & {{\cos \quad {\psi cos\theta}} - {\sin \quad {\psi sin\varphi sin\theta}}} & {{\cos \quad {\psi sin\theta}} + {\sin \quad {\psi sin\varphi cos\theta}}} \\{\sin \quad \varphi} & {{- \cos}\quad {\varphi sin\theta}} & {\cos \quad {\varphi cos\theta}}\end{bmatrix}}{F = \begin{bmatrix}1 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & {1/f}\end{bmatrix}}} & (10)\end{matrix}$

[0106] According to the equations (3), the coordinate conversion matrixcan be represented as (11): $\begin{matrix}{M = \begin{bmatrix}m_{1} & m_{2} & m_{3} \\m_{4} & m_{5} & m_{6} \\m_{7} & m_{8} & 1\end{bmatrix}} & (11)\end{matrix}$

[0107] From the equations (10) and (11), there is derived a relationrepresented by the equation (12): $\begin{matrix}{{{f = \sqrt{\frac{m_{1}^{2} + m_{4}^{2} - m_{2}^{2} - m_{5}^{2}}{m_{8}^{2} - m_{7}^{2}}}},{m_{8} \neq m_{7}}}{o\quad r}{{f = \sqrt{\frac{{{- m_{1}}m_{2}} - {m_{4}m_{5}}}{m_{6}m_{7}}}},{m_{6} \neq 0},{m_{7} \neq 0}}} & (12)\end{matrix}$

[0108] Consequently, the focal length estimation unit 112 estimates thefocal length by the equation (12), utilizing, as the parameters m₁ tom₇, those stored in the n-image integrating unit 111 corresponding tothe panning photographing. As there are plural sets of parameters, thereis employed the center value of the focal lengths f_(k) obtained fromsuch parameters. The estimated focal length f is retained in the n-imageintegrating unit 111.

[0109] The n-image combining unit 114 generates a combined image,utilizing the generated parameter f (step S42). As in the secondembodiment, the combined image is subjected to a cylindrical mappingconversion to generate a combined image of n images as shown in FIG. 13,and the combined image is written into the main image memory 113.

[0110] On the other hand, if the step S40 identifies 360° combination,there is executed estimation of the coordinate conversion parametersbetween the first and n-th images, since they mutually overlap (stepS43). This operation can be achieved by reading the first and n-thimages from the main image memory 113 and executing the aforementionedprocess in the combined image process unit 1. The information such asthe generated parameters is stored in the n-image integrating unit 111.

[0111] The focal length estimation unit 112 estimates the focal length(step S44). FIG. 18 shows the viewing angle ω_(k) in each image, whichis obtained from the following equation (13):

ω_(k)=tan⁻¹(x _(k) /f)  (13)

[0112] wherein x_(k) is obtained by determining the center line S of theoverlapping area between the images, as shown in FIG. 19. The n-imageintegrating unit 111 determines x_(k) by calculating the overlappingarea between the images based on the coordinate conversion parameters.

[0113] Since the viewing angle of 360° is covered by n images, the sumof ω₁ to ω_(n) becomes 360° (2π), and this relationship is representedby the equation (14):

2π−Σ{(2 tan⁻¹(x _(k) /f)}=0  (14)

[0114] The focal length is estimated by solving the equation (14) forexample by the Newton method. The estimated focal length is stored inthe n-image integrating unit 111. Also the n-image integrating unit 111executes cylindrical mapping conversion of the coordinates of thecorresponding points between the images, based on the focal length f,and generates parameters according to the equations (2), utilizing thusconverted coordinates of the corresponding points. The generatedparameters are further converted into values based on a reference image(for example image 1) for use in the combination of all the images.

[0115] In this manner the focal length f can be automatically obtained,also in the second embodiment, solely from the image information.

[0116] Then there is executed a conversion process for providing asmooth joint between the first and n-th images as a result of 360°combination (step S45). FIG. 20 shows the combined image after suchconversion process. Without the conversion process, the reference linesset respectively in the first and n-th images are mutually displaced.

[0117]FIG. 21 shows the combined image after such conversion process,wherein the reference lines respectively set in the first and n-thimages mutually coincide. FIG. 22 schematically shows the conversionprocess.

[0118] As shown in FIG. 22, a start point P₁ and an end point P_(N) ofthe entire circumference are set, and the images are so converted thatthe reference lines of the images 1 and n mutually coincide at the pointP_(N). In the illustrated case, the reference lines have a mutuallyaberration D at the point P_(N), while H pixels are present between thestart point P₁ and the end point P_(N), so that the write-in coordinatein the vertical direction is shifted by a value dν(x), given by thefollowing equation (15), taking the start point P₁ as the start point ofthe combined image in the horizontal direction: $\begin{matrix}{{{v(x)}} = {\frac{D}{H}x}} & (15)\end{matrix}$

[0119] The n-image integrating unit 111 generates and retains thecoefficient D/H in the equation (15), and the n-image combining unit 114generates the combined image (step S46). In this operation, it generatesthe pixel values of the combined image by executing the cylindricalmapping conversion according to the equations (8) and the conversionaccording to the equation (15) and stores the generated pixel value inthe main image memory 113.

[0120] The aberration in the image can thus be made inconspicuous, bydistorting the entire image with a constant rate in the panningdirection.

[0121]FIG. 23 schematically shows the combined image thus generated,which is displayed on display unit not shown. The above-describedprocess allows to generate an automatically mapped cylindrical imagefree from aberration over the entire circumference of 360°. FIG. 24shows the generation of the cylindrical image, mapped over the entirecircumference of 360°.

[0122]FIG. 25 is a block diagram showing a computer system realizing animage combination apparatus of a fourth embodiment of the presentinvention. In the fourth embodiment, a program embodying the imagecombination process of the foregoing first embodiment is recorded on arecording medium and is rendered operable on the computer system.

[0123] In FIG. 25, there are shown a memory 120 for storing variousdata, a CPU 121, a reproduction unit 122 for reproducing the informationof a recording medium, a display unit 123, and a recording medium 124.

[0124]FIG. 26 is a flow chart showing the image combination sequence inthe fourth embodiment. A corresponding program is stored in therecording medium 124 and is executed by the CPU 121. At first the userselects images to be used in combination (step S51), and the user setsthe arrangement of the images (step S52). Based on such arrangement, twoneighboring images are read into the memory 120 (step S53).

[0125] From thus read images, corresponding points are extracted (stepS54), and the photographing method is discriminated from the informationof the corresponding points (step S55). Based on the discriminatedphotographing method, an optimum combining method is set and thecombination parameters are generated (step S56).

[0126] Two images are combined utilizing the combination parameters(step S57), and the combined image is displayed (step S58). Then thereis discriminated whether the combinations of the neighboring two imageshave been completed (step S59).

[0127] If not, the sequence returns to the step S53 to repeat thecombination of two images, but, if completed, the parameters areconverted into values relative to the reference image. All the imagesare combined, based on thus converted combination parameters (step S61).The process in each step is similar to that explained in the foregoingembodiments.

[0128] The recording medium can be composed not only of a memory elementsuch as ROM but also of any other memory device such as a floppy disk, ahard disk, an optical disk, a magnetooptical disk, a CD-ROM, a CD-R, aDVD, a magnetic tape or a non-volatile memory card.

[0129] Many widely different embodiments of the present invention may beconstructed without departing from the spirit and scope of the presentinvention. It should be understood that the present invention is notlimited to the specific embodiments described in the specification,except as defined in the appended claims.

What is claimed is:
 1. An image processing method comprising: an inputstep of entering plural images; and a selection step of selecting anappropriate image combining method among plural image combining methods,utilizing said plural images.
 2. An image processing method according toclaim 1, further comprising: an evaluation step of evaluating imagecombination based on at least one of said plural image combiningmethods; wherein said selection step selects the image combining methodaccording to the result of said evaluation.
 3. An image processingmethod according to claim 2, further comprising: an extraction step ofextracting corresponding points on an overlapping area of said pluralimages.
 4. An image processing method according to claim 3, wherein saidevaluation step is adapted to calculate a combination error generated incase of image combination based on said corresponding points and whereinsaid selection step is adapted to evaluate the image combining method bycomparing said combination error with a predetermined value.
 5. An imageprocessing method according to claim 3, wherein said evaluation step isadapted to calculate combination errors generated in case of imagecombinations respectively by at least two image combining methods basedon said corresponding points and wherein said selection step is adaptedto evaluate the image combining method by comparing the magnitude ofsaid combination errors.
 6. An image processing method according toclaim 1, wherein said plural image combining methods at least includethose corresponding to a translation photographing method in whichimages are taken with a change in the photographing position but withouta change in the photographing direction, and a panning photographingmethod in which images are taken with a change in the photographingdirection but with a change in the photographing position.
 7. An imageprocessing method according to claim 6, wherein the image combiningmethod corresponding to said panning photographing method utilizesimages obtained by coordinate conversion from said plural images onto avirtual cylindrical plane or a virtual spherical plane having a radiusequal to the photographing focal length.
 8. An image processing methodaccording to claim 6, wherein the image combining method correspondingto said panning photographing method includes a correction step adapted,in case of generating a 360° panoramic image, to correct the imageaberration generated in the 360° panoramic image generated by combiningthe both ends of the combined image, by distorting the entity of saidcombined image.
 9. A recording medium storing a computer readableprogram for causing a computer to execute a process comprising: an inputstep of entering plural images; and a selection step of selecting anappropriate image combining method among plural image combining methods,utilizing said plural images.
 10. A recording medium according to claim9, wherein said process further comprises: an evaluation step ofevaluating image combination based on at least one of said plural imagecombining methods; wherein said selection step selects the imagecombining method according to the result of said evaluation.
 11. Arecording medium according to claim 10, wherein said process furthercomprises an extraction step of extracting corresponding points on anoverlapping area of said plural images.
 12. A recording medium accordingto claim 11, wherein said evaluation step is adapted to calculate acombination error generated in case of image combination based on saidcorresponding points and wherein said selection step is adapted toevaluate the image combining method by comparing said combination errorwith a predetermined value.
 13. A recording medium according to claim11, wherein said evaluation step is adapted to calculate combinationerrors generated in case of image combinations respectively by at leasttwo image combining methods based on said corresponding points andwherein said selection step is adapted to evaluate the image combiningmethod by comparing the magnitude of said combination errors.
 14. Arecording medium according to claim 9, wherein said plural imagecombining methods at least include those corresponding to a translationphotographing method in which images are taken with a change in thephotographing position but without a change in the photographingdirection, and a panning photographing method in which images are takenwith a change in the photographing direction but with a change in thephotographing position.
 15. A recording medium according to claim 14,wherein the image combining method corresponding to said panningphotographing method utilizes images obtained by coordinate conversionfrom said plural images onto a virtual cylindrical plane or a virtualspherical plane having a radius equal to the photographing focal length.16. A recording medium according to claim 14, wherein the imagecombining method corresponding to said panning photographing methodincludes a correction step adapted, in case of generating a 360°panoramic image, to correct the image aberration generated in the 360°panoramic image generated by combining the both ends of the combinedimage, by distorting the entity of said combined image.
 17. An imageprocessing apparatus comprising: input means for entering plural images;and selection means for selecting an appropriate image combining methodamong plural image combining methods, utilizing said plural images. 18.An image processing apparatus according to claim 17, further comprising:evaluation means for evaluating image combination based on at least oneof said plural image combining methods; wherein said selection meansselects the image combining method according to the result of saidevaluation.
 19. An image processing apparatus according to claim 18,further comprising: extraction means for extracting corresponding pointson an overlapping area of said plural images.
 20. An image processingapparatus according to claim 19, wherein said evaluation means isadapted to calculate a combination error generated in case of imagecombination based on said corresponding points and wherein saidselection means is adapted to evaluate the image combining method bycomparing said combination error with a predetermined value.
 21. Animage processing apparatus according to claim 19, wherein saidevaluation means is adapted to calculate combination errors generated incase of image combinations respectively by at least two image combiningmethods based on said corresponding points and wherein said selectionmeans is adapted to evaluate the image combining method by comparing themagnitude of said combination errors.
 22. An image processing apparatusaccording to claim 17, wherein said plural image combining methods atleast include those corresponding to a translation photographing methodin which images are taken with a change in the photographing positionbut without a change in the photographing direction, and a panningphotographing method in which images are taken with a change in thephotographing direction but with a change in the photographing position.23. An image processing apparatus according to claim 22, wherein theimage combining method corresponding to said panning photographingmethod utilizes images obtained by coordinate conversion from saidplural images onto a virtual cylindrical plane or a virtual sphericalplane having a radius equal to the photographing focal length.
 24. Animage processing apparatus according to claim 22, wherein the imagecombining method corresponding to said panning photographing methodincludes correction means adapted, in case of generating a 360°panoramic image, to correct the image aberration generated in the 360°panoramic image generated by combining the both ends of the combinedimage, by distorting the entity of said combined image.
 25. An imageprocessing method comprising: an input step of entering plural images;an extraction step of extracting corresponding points in an overlappingarea of said plural images; and a selection step of selecting an imagecombining method based on said corresponding points.
 26. An imageprocessing method according to claim 25, further comprising: anevaluation step of evaluating image combination based on at least one ofsaid plural image combining methods; wherein said selection step selectsthe image combining method according to the result of said evaluation.27. An image processing method according to claim 25, wherein saidplural image combining methods at least include those corresponding to atranslation photographing method in which images are taken with a changein the photographing position but without a change in the photographingdirection, and a panning photographing method in which images are takenwith a change in the photographing direction but with a change in thephotographing position.
 28. An image processing method according toclaim 26, wherein said evaluation step cylindrical mapping conversion orspherical mapping conversion.
 29. An image processing method accordingto claim 25, wherein said image combining method utilizes affinconversion or projection conversion.
 30. An image processing methodaccording to claim 25, further comprising a correction step adapted, incase of generating a 360° panoramic image, to correct the imageaberration generated in the 360° panoramic image generated by combiningthe both ends of the combined image, by distorting the entity of saidcombined image.